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Introduction 

There  is  growing  interest  in  the  healthcare  field  for  telemedicine,  the  use  of  teleconferencing  technologies 
(bidirectional  video  and  audio)  to  allow  consultations  between  doctor  and  patient  located  in  different 
places.  Telemedicine  reduces  travel  times  for  physicians  and  patients,  and  allows  specialized  healthcare  to 
reach  locations  that  might  otherwise  be  underserved.  With  the  growing  acceptance  of  telemedicine  comes 
the  possibility  of  other  types  of  remote  medical  intervention,  such  as  remote  surgery.  Remote  surgery, 
where  one  or  more  members  of  the  surgical  team  are  remotely  located,  is  presently  in  its  infancy.  One  of 
the  prime  motivators  of  this  concept  has  been  the  United  States  Department  of  Defense,  who  saw  the 
potential  for  far-forward  battlefield  medical  intervention  without  placing  highly  valuable  surgeons  in 
danger.  Presently,  implementations  consist  of  research  systems  in  which  a  remote  surgeon  instructs 
(teleoperates)  a  robot  for  suturing  or  for  micro-inspection. 

We  are  constructing  the  Integrated  Remote  Neurosurgical  System  (IRNS),  a  remotely-operated 
nemosurgical  microscope  with  high-speed  communications  and  a  sxu-geon-accessible  user  interface.  The 
BRNS  will  allow  high  quality  bidirectional  mentoring  in  the  neurosurgical  suite,  allowing  a  remote 
neurosurgeon  to  lend  expertise  to  the  Operating  Room-based  (OR)  neurosurgical  team. 

The  IRNS  uses  a  generic  microscope/transport  model,  allowing  the  system  to  be  useful  for  remote 
operation  of  a  general  class  of  operating  room  robots  and  video  sources.  In  our  case,  the  equipment  is  the 
Carl  Zeiss  [1]  MKM  surgical  microscope/transport,  with  attached  Zeiss  OPMI  stereo  microscope  (Figure 
1).  SuMIT  (the  Surgical  Manipulator  Interface  Translator)  performs  translation  from  generic  commands 
to  hardware-specific  transport  commands  in  the  OR.  A  SuMIT  interface  is  planned  for  each  device  to  be 
controlled  by  the  IRNS. 

To  the  remote  surgeon,  the  most  important  aspect  of  the  IRNS  is  the  feeling  of  being  present  in  the  OR 
during  the  surgery.  It  is  this  sensation  of  telepresence  that  enables  the  remote  surgeon  to  comfortably  and 
confidently  control  the  remote  microscope  and  interact  smoothly  with  the  OR  team.  To  be  successful, 
telepresence  systems  must  place  a  low  cognitive  load  on  the  user  by  allowing  him  to  interact  with  the 
system  in  a  natural  and  familiar  maimer.  For  example,  the  use  of  bidirectional  audio  allows  the  users  of  a 
telepresence  system  to  communicate  verbally  as  they  would  in  person.  The  addition  of  bidirectional  video 
adds  the  information  contained  in  facial  expressions  and  gestures.  This  type  of  feedback  is  known  to 
improve  the  interactivity  of  collaborative  systems[2].  If  other  physical  artifacts  are  present  in  the  system, 
such  as  input  devices  with  which  the  user  interacts,  they  should  be  familiar  in  appearance  and  operation  to 
those  used  under  normal  circumstances.  In  this  manner  our  telepresence  system  can  be  usefiil  and 
acceptable  to  the  surgeon.  As  we  will  show,  we  have  placed  a  strong  emphasis  on  telepresence  in  the 
IRNS. 


5 


Figure  1.  The  MKM  microscope  transport 

The  IRNS  remote  workstation  incorporates  image-based  surgical  planning  capabilities  and  can  display 
real-time  video  from  both  the  microscope  and  an  overhead  video  camera.  The  overhead  camera  shows  the 
remote  surgeon  the  general  layout  of  the  OR  and  the  relative  locations  of  the  OR  team,  the  patient,  and  the 
robot.  The  IRNS  also  incorporates  the  ability  to  remotely  control  the  microscope  position,  zoom,  and 
focus.  The  remote  workstation  includes  several  methods  for  microscope  control.  The  primary  interaction 
method  is  a  force-reflecting  handgrip  mounted  on  a  6-degree-of-freedom  robot  that  allows  the  remote 
smgeon  to  intuitively  position  the  microscope.  Other  features  include  bidirectional  audio,  image 
archiving,  voice  recognition,  bidirectional  whiteboarding,  and  an  interface  to  our  presmgical  stereotactic 
plaiming  system. 

Real-time  simulation  of  the  microscope  transport  is  provided  by  the  commercially  available  Telegrip 
simulation  package  from  Deneb  Robotics[3].  Telegrip  is  used  to  produce  a  realistic  view  of  the  OR  robot 
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as  it  is  controlled  from  the  remote  workstation.  The  simulation  permits  pre-surgical  simulation  and 
training  for  surgeons  without  access  to  an  actual  microscope  transport  system. 

The  components  of  the  IRNS  are  integrated  using  Asynchronous  Transfer  Mode  (ATM)[4]  switching  to 
provide  low  latency  data  transfer.  A  registration  system  has  been  developed  to  ensure  precise  monitoring 
of  the  patient  and  microscope. 

The  IRNS  provides  an  opportunity  to  assess  the  benefits  of  remote  surgical  mentoring  and  to  gain 
technical  expertise  in  teleoperation  of  surgical  devices.  By  integrating  high  performance  workstations  and 
microscope  carriers,  the  system  demonstrates  the  usefulness  of  communications  and  computing  resources 
in  improving  access  to  specialized  surgical  facilities.  This  research,  along  with  the  more  sophisticated 
systems  that  will  follow,  will  serve  as  a  foundation  and  testing  platform  for  extending  the  surgeon’s  skills 
without  regard  to  time  zone  or  geographic  boimdaries. 

This  document  will  describe  the  IRNS  in  its  current  state,  including  the  system  architecture, 
communications  architecture,  safety  guidelines,  user  interface,  and  hardware.  We  will  then  discuss  the 
future  research  and  development  goals  for  the  IRNS. 


IRNS  System  Architecture 

Overview 

The  IRNS  consists  of  two  worksites:  the  operating  room  and  the  remote  workstation.  They  are  connected 
via  an  ATM  network  ( Figure  2).  Each  worksite  hosts  elements  such  as  workstations  and  cameras. 

Locally,  these  elements  are  interconnected  by  ATM,  Ethernet,  or  RS-232  connections,  and  are  ultimately 
linked  by  a  site  workstation.  The  ATM  link  is  responsible  for  carrying  all  video,  audio,  and  control  data. 
Currently,  the  remote  workstation  may  control  a  simulated  MKM  (in  Telegrip)  or  a  Robotics  Research 
(RR)  1607,  which  is  installed  in  an  OR  mockup  in  our  lab.  The  RR1607  is  interfaced  to  a  servo- 
controlled  charge-coupled  device  (CCD)  camera  that  is  analogous  to  the  OPMI  stereo  microscope.  The 
MKM  teleoperation  interface  is  currently  under  development  by  Carl  Zeiss.  The  ATM  network,  room 
camera  software,  audio  communications,  and  user  interface  are  all  present  in  the  current  implementation 
of  the  system. 

A  typical  scenario  for  use  of  the  IRNS  consists  of: 

•  microscope  setup  and  positioning, 

•  session  startup  and  connection  to  remote  workstation, 

•  preliminary  consultation  between  OR  and  remote  physicians, 

•  commencement  of  surgery  with  remote  physician  observing  and  providing  expertise, 

•  handoff  of  control  of  microscope  between  OR  and  remote  physicians, 

•  remote  control  of  microscope,  including  gross  and  fine  motion,  and 

•  session  shutdown. 

Each  task  in  this  list  is  now  under  development  or  testing  in  the  IRNS.  Some,  such  as  the  control  handoff 
and  session  shutdown,  are  in  early  stages  of  development,  and  will  be  completed  during  the  remainder  of 
the  project.  We  will  now  examine  each  item  in  greater  detail. 

Microscope  setup  and  positioning  involves  both  microscope-specific  tasks  (such  as  driving  the  MKM 
transport  into  the  OR  and  placing  in  position)  and  IRNS-specific  tasks  such  as  estabhshing  the 
communications  link  between  the  microscope  and  the  OR  workstation.  Microscope-specific  tasks  are 
performed  first.  For  the  MKM,  setup  includes  physical  positioning  of  the  transport  and  the  workstation  in 
the  OR,  making  the  appropriate  cabling  connections,  booting  of  the  MKM  workstation,  and  executing  the 
MKM  control  software,  and  executing  the  MKM-IRNS  link  software.  This  software  provides  the  IRNS 
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generic  interface  to  the  MKM  controller.  Once  the  MKM-IRNS  link  software  is  running,  the  system  is 
controlled  generically  using  the  IRNS. 

The  IRNS  obtains  the  current  robot  position  and  the  patient  registration  transformation  from  the  MKM 
using  the  MKM-IRNS  link  software.  This  information  is  used  for  image-guided  control  of  the  microscope. 

The  remote  workstation  is  connected  to  the  OR  workstation  next.  The  method  used  to  establish  the 
communications  link  will  be  described  in  later  sections;  it  is  transparent  to  the  users  of  the  system  in  any 
case.  Startup  of  the  remote  workstation  software  is  performed  simply  by  execution  of  a  single  UNIX 
conunand.  Once  the  software  is  running,  the  remote  surgeon  will  typically  specify  a  patient’s  image  data 
set  to  load. 

Once  the  connection  between  sites  has  been  established,  the  OR  and  remote  singeons  may  greet  each  other 
verbally  and  visually.  Each  can  see  the  other  on  their  workstation  screens,  and  speak  normally  as  if  in  a 
face-to-face  conversation.  A  brief  discussion  may  take  place  about  the  surgical  plan  or  specific  details  of 
the  case.  Once  surgery  begins,  the  OR  surgeon  may  use  the  microscope  normally,  repositioning  it  and 
viewing  the  surgical  site  through  it. 

Handoff  of  control  to  the  remote  surgeon  is  designed  to  be  unobtrusive  and  positive.  That  is,  both  sites 
know  that  control  has  been  handed  off,  and  no  undesirable  or  unsafe  motions  ensue.  The  handoff  is 
initiated  verbally  by  the  remote  surgeon.  That  is,  he  simply  asks  for  control,  and  the  OR  surgeon  may 
agree  or  disagree  to  the  handoff.  If  it  is  agreeable  to  the  OR  surgeon,  a  foot  pedal  or  screen  control  is 
toggled  in  the  OR.  This  causes  a  signal  to  be  issued  to  the  remote  site  informing  the  surgeon  that  he  may 
take  control.  A  screen  control  is  then  toggled  by  the  remote  surgeon,  giving  him  control  of  the  system.  At 
any  time,  control  may  be  taken  back  by  the  OR  -  a  toggle  is  performed  in  the  OR  giving  control  back  to 
the  OR  surgeon.  The  control  handoff  is  then  indicated  on  the  remote  workstation  screen. 

The  IRNS  provides  the  remote  surgeon  with  the  ability  to  perform  gross  robot  motion,  fine  robot  motion, 
microscope  control,  image  retrieval,  and  image  capture  and  annotation  facilities.  The  remote  surgeon  has 
three  methods  of  controlling  the  remote  robot:  image  plane  translation,  image  guided  motion  (using 
preoperative  MRI/CT  images  registered  to  the  patient),  or  analogical  motion  using  a  force  reflecting  input 
device.  The  surgeon  may  generate  either  gross  or  fine  motions  using  the  force  reflecting  input  device. 

Session  shutdown  is  performed  after  verbal  agreement  by  both  parties.  The  remote  surgeon  then  toggles  a 
control  on  his  workstation  indicating  that  the  connection  is  to  be  broken.  In  the  OR,  a  message  is 
displayed  on  the  OR  workstation  indicating  that  the  remote  surgeon  is  no  longer  present,  but  OR 
operation  is  not  affected.  Shutdown  in  the  OR  is  performed  by  exiting  the  IRNS  software  and  shutting 
down  the  microscope  in  the  normal  fashion. 
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Figure  2.  Block  diagram  of  the  Integrated  Remote  Neurosurgical  System 


Communications  Design 

Asynchronous  Transfer  Mode  (ATM)  networking  is  an  emerging  communications  technology  in  which 
fixed-length  “cells”  of  data  are  transferred  at  speeds  of  over  2.3  Gbits/sec[5].  ATM  supports  different 
quality-of-service  (QoS)  levels  depending  on  traffic  types,  and  fimctions  similarly  over  local  and  wide  area 
networks.  ATM  cells  can  be  switched  through  inter-exchange  carrier  networks  and  carried  over  earth 
satellite  links,  making  it  ideal  for  remote  delivery  of  critical  teleoperation  applications.  The  primary 
advantage  of  ATM  is  that  it  was  specifically  designed  for  simultaneous  data,  voice  and  video  transmission 
-  precisely  the  characteristics  required  for  telemedicine. 

We  have  implemented  a  private  ATM  network  for  carrying  digital  as  well  as  audio/video  data.  The 
network  is  composed  of  three  Newbridge  Networks  36150  ATM  switches,  two  four  port  switches  and  one 
eight  port  model.  All  connections  both  between  the  switches  and  to  the  network  end  points  are 
150Mbit/sec  OC3.  The  long  haul  connection  between  the  remote  workstation  and  the  hospital  is  made 
using  single  mode  fiber.  All  other  ATM  devices  are  physically  connected  using  multi-mode  fiber.  Three 
high  performance  UNIX  workstations  are  attached  to  this  network,  two  in  the  lab  and  one  in  the  hospital 
OR.  Each  workstation  includes  video  and  audio  sources  and  monitors.  These  are  connected  to  the  ATM 
switches  via  coaxial  cables,  where  they  are  digitized  using  JPEG  ATM  video  compression  boards.  This 
system  is  used  to  transport  a  combination  of  live  video,  audio  and  digital  control  data  between  the  various 
endpoints  of  the  network.  See  Figure  3. 

For  transmission  of  image  data  and  control/feedback  information,  both  locally  and  remotely,  we  use 
TelRIP[6],  a  general  purpose  data  exchange  system.  TelRIP  uses  a  data-centered  approach  to  modularity, 
meaning  that  programs  communicate  with  one  another  by  specifying  the  fypes  of  data  they  wish  to  send  or 
receive.  We  have  developed  a  standardized  set  of  TeUUP  data  objects  that  are  used  to  transmit  generic 
information,  such  as  microscope  settings,  positions,  and  localizer  positions.  By  using  TelRIP,  the  IRNS 
has  been  developed  as  a  set  of  discrete  peer  applications  that  run  largely  asynchronously.  An  advantage  is 
that  our  processes  can  continue  to  run  without  knowing  the  state  of  other  processes  on  the  network.  This 
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Remote  Workstation 


Figure  3.  ATM  network  layout 


approach  also  allows  us  to  standardize  the  interfaces  to  devices  and  controls  without  predetermining  the 
exact  implementation  of  each  interface. 

Currently,  the  OC3  ATM  link  from  the  OR  to  the  remote  workstation  in  our  lab  is  incomplete.  We  are 
expecting  the  final  connections  to  be  completed  by  30  September,  1997.  Until  this  time,  we  continue  to 
experiment  using  our  OR  mockup  in  the  lab.  The  OR  mockup  is  linked  via  ATM  to  the  remote 
workstation.  Presently,  all  video  signals  are  transmitted  via  ATM  using  hardware  JPEG  compression. 
Data  signals  are  transmitted  using  Ethernet  running  on  lOBase-T  connections.  This  will  be  upgraded  to 
ATM  by  30  September,  1997.  Audio  connections  will  be  run  over  the  ATM  at  the  same  time,  after  audio 
wiring  is  run  from  the  mock  OR  and  remote  workstations  to  our  ATM  switch. 


OR  Worksite 

The  OR  worksite  consists  of: 

•  A  surgical  stereo  microscope  mounted  on  a  robotic  transport, 

•  One  or  more  ‘bird’s  eye’  view  cameras  on  pan/tilt  heads, 

•  An  optical  registration  system,  and 

•  A  workstation  to  display  and  annotate  pre-surgical  and  inter-operative  medical  imagery. 

Live  video  from  both  the  stereo  microscope  and  the  OR  view  camera(s)  is  routed  through  JPEG 
compression  hardware  before  transmission  across  the  ATM  link  and  is  subsequently  decompressed,  again 
in  hardware,  at  the  remote  location.  The  OR  bird’s  eye  view  cameras  are  included  to  give  the  remote 
surgeon  feedback  on  the  situation  in  the  OR. 

The  optical  registration  system  calculates  the  relative  positions  of  the  robot  base  and  patient  by  sensing 
targets  that  are  placed  on  them.  Locations  of  these  objects  are  represented  internally  as  a  series  of 
coordinate  systems.  Using  these  locations,  as  well  as  other  information  gathered  by  the  system,  we  can 
monitor  the  location  of  the  patient  relative  to  the  microscope.  Additionally,  we  can  calciriate  the  spatial 
relationship  between  the  patient  and  the  pre-operative  image  database  created  using  magnetic  resonance 
(MR)  or  computed  tomography  (CT).  This  allows  the  system  to  move  the  microscope  to  a  position  relative 
to  objects  indicated  in  the  images,  or  according  to  the  pre-operative  plan.  Registration  of  the  patient  to 
images  is  performed  using  a  manual  procedure  of  locating  fiducials  that  are  attached  to  the  patient’s  skin 
prior  to  imaging. 

In  our  OR  mockup,  optical  registration  is  performed  using  an  Image  Guided  Technologies’  Flashpoint 
5000  [7].  The  Flashpoint  consists  of  three  cameras  in  a  rigid  frame  and  several  dynamic  reference  frames 
(DRF’s)  containing  infrared  light-emitting  diodes  (LED’s).  The  DRF’s  are  rigidly  attached  to  the  patient 
head  frame  and  to  the  robot.  By  measuring  the  spatial  relationships  of  the  LED’s  on  the  DRF’s  from  the 
three  cameras,  the  Flashpoint  determines  the  position  and  orientation  of  the  patient  DRF  with  respect  to 
the  robot  DRF.  Once  this  relationship  is  known,  a  point-to-point  registration  of  fiducials  is  performed  to 
calculate  the  exact  transformation  from  the  image  base  to  the  patient.  The  registration  is  performed  using 
a  probe  instrumented  in  the  same  maimer  as  the  DRF’s.  As  the  user  indicates  a  series  of  fiducials  on  the 
patient,  the  Flashpoint  returns  the  position  of  the  fiducial  with  respect  to  the  patient  DRF.  As  each  point  is 
located,  the  user  indicates  the  same  fiducial  in  the  image  database.  Once  this  process  is  complete,  the 
system  automatically  calculates  the  relationship  between  the  robot  base  frame,  patient,  and  image  base. 

In  the  OR,  the  MKM  uses  a  built-in  optical  registration.  The  microscope  is  guided  to  focus  on  each 
fiducial  in  turn,  and  the  corresponding  fiducial  is  indicated  in  the  MR  or  CT  images.  After  all  fiducials 
have  been  manually  located,  the  system  calculates  the  transform  between  the  robot  base  frame,  patient, 
and  image  base. 

The  OR  workstation  can  display  views  from  the  stereo  microscope,  pre-surgical  MR  and  CT  images,  and 
3D  surgical  planning  images.  Whiteboarding  (described  below  in  the  User  Interface  section)  is  supported, 
allowing  the  OR  team  and  remote  surgeon  to  communicate  by  drawing  on  the  video  images.  Users  in  the 
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OR  can  interactively  slice  through  patient  data,  magnify  images,  and  view  a  rendered  volume  of  the  brain 
using  thumbwheel-style  widgets  beneath  the  windows.  In  addition,  verbal  communication  between  the  OR 
and  the  remote  site  is  provided  through  microphones  and  ATM-linked  audio. 


Remote  Worksite 

The  remote  worksite  consists  of  a  graphics  workstation  providing: 

•  Microscope  control, 

•  Robotic  transport  control, 

•  Video  from  the  microscope  and  OR  view  cameras, 

•  Whiteboarding, 

•  Image  (snapshot)  archiving  and  annotation  (using  graphics  and  audio), 

•  Voice  recognition  commands, 

•  Force-reflecting  handcontroller  with  MKM  handgrip, 

•  Presurgical  image  display,  including  a  calculated  oblique  slice  orthogonal  to  the  view  axis, 

•  Three-dimensional  segmented  brain  model  display,  and 

•  Surgical  planning  facilities. 

Microscope  controls  that  are  extended  to  the  remote  workstation  include  manual  focus  and  zoom 
adjustments  as  well  as  the  automatic  focusing  control.  These  controls  can  be  activated  by  using  verbal 
conunands  to  the  system’s  speech  recognition  software,  Speech  Systems  Phonetic  Engine  500[8]  or  by 
pressing  the  appropriate  controls  on  the  handgrip.  See  Figure  4. 


Figure  4.  The  remote  workstation 

Microscope  transport  positioning  control  can  be  performed  in  three  ways: 

•  Graphically  -  The  surgeon  can  manipulate  the  graphical  user  (GUI)  to  position  the  microscope. 

•  Verbally  -  The  surgeon  can  issue  commands  to  the  microscope  by  speaking. 

•  Analogically  -  The  surgeon  can  use  a  force-reflecting  handcontroller  to  position  the  microscope. 

Facilities  on  the  layout  of  the  GUI  allow  the  surgeon  to  control  the  position  of  the  microscope  in  image 
space  or  in  relation  to  the  patient.  In  the  former  case,  the  surgeon  may  translate  the  microscope  and  zoom 
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using  arrow  buttons  on  the  edges  of  the  video  window.  In  this  mode,  the  orientation  of  the  microscope 
remains  constant,  and  motion  is  performed  as  relative  translations  in  the  “tool  frame”  of  the  robot.  The 
control  buttons  can  be  activated  either  by  using  the  mouse  or  by  pressing  the  touchscreen. 

In  addition,  speech  recognition  software  allows  the  remote  surgeon  to  control  panning  and  zooming  of  the 
microscope  simply  by  speaking  “pan  left”,  “zoom  in”,  etc.  Most  of  the  GUI  functions  now  have  verbal 
equivalents,  such  as  panning  the  microscope  and  taking  snapshots  of  the  current  microscope  view.  To 
issue  voice  commands,  the  surgeon  presses  a  foot  switch  and  speaks  normally.  The  remote  workstation 
includes  a  microphone  attached  to  the  face  of  the  monitor,  but  a  headset  may  be  used  in  noisy 
environments.  The  system  is  speaker  independent.  Response  has  been  positive  from  our  primaiy 
consulting  surgeon  and  casual  visitors  to  our  lab;  this  technique  of  interaction  is  continually  being 
enhanced  to  include  more  commands. 

To  position  the  microscope  relative  to  the  patient,  the  surgeon  indicates  a  target  point  (center  of  interest) 
on  the  three-dimensional  segmented  brain  display,  and  an  “entry  point”.  The  entry  point  is  a  point 
situated  along  the  view  axis  of  the  microscope.  Where  the  entry  point  is  located  on  the  axis  is 
unimportant;  the  point  is  used  simply  to  calculate  a  unit  vector  for  calculation  of  the  microscope 
orientation.  The  final  position  of  the  microscope  is  calculated  by  multiplying  the  unit  vector  by  a  fixed 
view  distance.  The  view  distance  is  currently  a  system  parameter  set  to  600mm;  however,  we  intend  to 
make  the  view  distance  variable  based  on  surgeon  preference. 

Analogical  motion  of  the  microscope  is  accomplished  using  a  tabletop  robot  (Puma  260)  with  a  MKM 
handgrip  attached.  The  MKM  handgrip  provides  force  and  moment  readings  at  20  Hz  as  the  surgeon 
pushes,  pulls,  and  twists  the  device.  These  readings  are  fed  into  a  Pentium-based  personal  computer  in 
which  a  real-time  control  loop  moves  the  Puma  to  minimize  the  forces  and  moments.  Thus,  the  system 
remains  in  equilibrium  when  the  surgeon  is  not  holding  the  handgrip;  when  the  surgeon  applies  forces  to 
the  handgrip,  the  robot  moves  to  return  the  force  value  to  zero.  The  relative  motion  is  scaled  and  applied 
to  the  OR  robot  as  a  position  control  command.  By  changing  the  force  control  equations,  we  can 
approximate  dynamic  effects  such  as  inertia. 

Because  the  force  control  system  uses  the  MKM  handgrip,  with  which  the  surgeon  may  already  be 
familiar,  and  emulates  a  free  floating  body,  training  time  is  minimized.  However,  we  continue  to  fine-tune 
parameters  in  the  force  controller  to  improve  the  control  stability,  avoid  singularities  in  the  robot 
kinematics,  and  limit  joint  and  cartesian  velocities.  The  low  sample  rate  of  the  MKM  handgrip  causes 
problems  for  smooth  motion  and  stability.  Force  control  applications  are  more  typically  servoed  at  IKHz. 
We  have  experimented  with  a  JR3  force/moment  sensor  positioned  between  the  robot  and  handgrip.  This 
sensor  provides  data  at  IKHz.  It  gave  better  results  than  the  MKM  sensor,  but  it  adds  weight  and  length  to 
the  overall  handgrip.  Another  difiiculty  with  the  force  control  system  is  the  high  fiiction  present  in  the 
Puma.  This  is  a  highly  nonlinear  effect  that  cannot  be  reasonably  modeled.  We  have  adopted  the  usual 
approach  to  friction  by  increasing  control  gains,  which  is  adversely  affecting  system  stability. 

For  added  safety,  we  have  implemented  a  force  reflection  component  in  the  Puma  controller.  Arbitrary 
geometric  volumes  may  be  designed  and  issued  to  the  real-time  controller,  which  will  restrict  the  Puma’s 
motion  to  remain  within  the  voliune.  For  example,  a  conical  volume  may  be  specified,  with  the  apex 
located  at  the  center  of  interest  on  the  patient,  and  the  axis  coincident  with  the  view  vector.  This  volume 
will  restrict  the  microscope  to  remain  within  a  reasonable  radius  of  the  original  view  vector.  As  the 
surgeon  pushes  the  handgrip  toward  the  (invisible)  edge  of  the  volume,  he  will  feel  no  resistance  until  the 
boundary  is  encountered.  At  the  boundary,  an  opposing  force  is  automatically  generated,  keeping  the 
handgrip  within  the  volume.  The  effect  is  quite  natural,  and  it  is  possible  to  ‘explore’  the  volume  and 
determine  its  basic  shape  without  a-priori  knowledge.  We  intend  to  improve  the  volume  design  features  so 
that  the  volume  may  be  automatically  generated  based  on  the  robot  and  patient  positions,  and  to  display 
the  volume  dynamically  in  Telegrip. 
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Prior  to  the  force  control  system,  we  have  also  experimented  with  ‘‘props”  [9]  real-world  objects  with 
magnetic  trackers  embedded  within.  There  are  two  props:  one  that  represents  the  microscope  (a  stylus) 
and  one  that  represents  the  patient’s  head  (a  doll’s  head).  By  monitoring  the  orientation  of  these  trackers, 
the  system  can  give  the  actual  microscope  the  same  orientation  with  respect  to  the  patient.  This  interface 
technique  has  been  in  use  at  our  lab  for  some  time  (on  a  prototype  surgical  planning  tool),  and  has 
received  enthusiastic  support  from  both  surgeons  and  residents  for  its  ease  of  use.  In  this  application,  it 
was  not  as  successful  due  to  the  difficulty  in  performing  fine  motions  and  restricting  dangerous  or 
impossible  robot  motion  commands. 

Feedback  from  the  OR  is  provided  in  the  form  of  video,  audio,  and  data.  A  graphical  display  of  the 
microscope’s  current  view  vector  is  shown,  to  give  the  surgeon  a  better  feel  for  the  spatial  configuration  of 
the  patient,  microscope,  and  transport.  The  remote  surgeon  is  presented  with  live  (30J[jps)  video  from  the 
microscope  and  the  0/R  room  camera  views.  Controls  on  the  window  allow  the  surgeon  to  pan  the  OR 
room  camera  in  4  directions  as  well  as  zoom. 

We  have  also  implemented  a  “point-click”  camera  control  interface[10].  This  technique  allows  the 
surgeon  to  simply  point  at  an  object  of  interest  in  the  room  view  and  the  camera  automatically  centers  that 
object  in  the  view.  The  surgeon  may  also  indicate  a  rectangular  region  (a  “zoom  box”)  in  the  view  and  the 
system  will  pan  and  zoom  to  align  the  edges  of  the  visible  image  with  the  edges  of  the  indicated  region. 
These  techniques  require  careful  calibration  of  the  pan/zoom  characteristics  of  the  camera. 

Snapshots  of  the  view  through  the  microscope  can  be  taken  and  saved  along  with  the  positioning 
information  necessary  to  return  the  transport  to  the  snapshot  view.  This  allows  the  remote  surgeon  to 
make  comparisons  over  the  course  of  a  procedure,  to  document  the  procedure  as  it  occurs,  or  to  use  the 
snapshots  as  a  visual  aid  during  consultations  with  the  OR  team.  To  return  the  microscope  to  the  position 
from  which  snapshots  were  taken,  commands  may  be  issued  via  the  graphical  user  interface  or  by  verbal 
command.  A  separate  window  normally  displays  the  most  recent  snapshot,  but  a  history  menu  of 
thumbnail  images  lets  the  user  select  an  older  snapshot  to  display.  Whiteboarding  is  supported  over  all 
snapshots.  When  a  snapshot  is  taken  at  either  site,  the  other  site  receives  a  copy  so  that  whiteboarding 
remains  in  synchronization. 

The  surgical  planning,  image  viewing,  and  audio  facilities  are  identical  to  those  on  the  0/R  workstation. 
Figure  5  and  Figure  6  show  screen  views  of  the  remote  workstation. 


Task  Analysis 

Feedback  from  surgeons  and  residents  has  been  sought  throughout  the  development  of  the  IRNS. 
Observations  of  surgeons  using  robotic  microscopes  have  been  conducted  at  the  University  of  Virginia 
hospital,  and  have  also  been  reviewed  on  video  logs.  These  observations  are  critical  in  determining  the 
current  deficiencies  in  the  usability  of  the  surgical  microscope  as  well  as  the  interaction  style  to  which  the 
siugeons  have  become  accustomed.  Our  system  can  correct  some  of  the  observed  problems  (operating 
table  movement,  jerky  motion  of  the  microscope)  because  of  the  increased  flexibility  afibrded  by  its  remote 
interface  and  registration  system.  Interviews  were  also  conducted  to  determine  what  major  functions 
surgeons  desire.  Features  which  came  out  of  this  process  include  speech  recognition  capabilities,  a 
simulated  view  of  the  microscope  (through  Telegrip),  the  oblique  slice  view,  the  surgical  planner 
interface  and  the  snapshot  facility. 

In  addition  to  the  design  criteria  specified  by  the  task,  current  research  concepts  from  the  areas  of  robotic 
control  paradigms  and  human-computer  interaction  were  included  in  the  design.  The  selection  of  input 
devices  and  the  graphical  user  interface  layout  and  interaction  style  were  influenced  in  part  by  previous 
research[ll,  12]. 
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In  the  area  of  robot  control,  we  are  working  to  exploit  control  strategies  that  are  adaptive  to  time-varying 
delays  and  guarantee  the  control  performance.  The  goal  to  develop  a  generalized  approach  to  bilateral 
controlled  teleoperators  over  an  ATM  network.  We  are  concentrating  on  source  traffic  characterization  in 
ATM  networks  and  the  design  of  bilateral  control  laws  in  presence  of  the  time-varying  delay.  The  work 
includes  the  bandwidth  scheduling,  the  estimation  of  end-to-end  transfer  delay  bound,  and  statistic 
analysis  of  the  delay  pattern.  We  discuss  this  analysis  beginning  on  page  16, 


Robot  Simulation 

Models  of  the  RR1607  and  the  MKM  have  been  developed  in  Telegrip  so  that  the  actions  of  the  robot 
could  be  simulated  when  it  was  not  actually  available.  The  MKM  simulation  includes  a  closed-form 
inverse  kinematics  module  linked  into  Telegrip  through  a  shared  library.  The  closed-form  solution  is  more 
rapid  than  the  iterative  one  and,  more  importantly,  is  capable  of  determining  the  singular  positions  within 
the  workspace  of  the  microscope  transport.  The  RR1607  model  is  controlled  using  either  the  built-in 
Telegrip  iterative  inverse  kinematics  solution  or  a  solution  we  have  developed.  It  is  not  possible  to  derive  a 
closed  form  solution  for  the  RR1607  because  of  the  redundancy  in  its  seven  degree-of-freedom  design. 
However,  it  was  necessary  to  develop  our  own  solution  so  it  could  be  installed  in  the  OR  mockup.  The 
simulation  models  can  be  manipulated  locally  through  the  Telegrip  UI  and  remotely  through  any  TCP/IP 
link.  Various  other  components  of  the  O/R  have  also  been  modeled  in  this  simulation  including  the 
camera,  optical  tracker  and  patient. 


Figure  5.  Screen  view  of  the  remote  workstation,  showing  the  snapshot  facility 
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Figure  6.  Screen  view  of  the  remote  workstation,  showing  the  image  viewers 

Issues  in  Bilateral  Control  of  Teleoperator  over  ATM  Networks 

This  section  describes  in  more  detail  our  work  on  developing  advanced  control  techniques  over  ATM 
networks.  The  work  is  critical  to  the  IRNS,  since  it  is  the  basis  of  our  analogical  control  system  (the  Puma 
desktop  robot).  However,  the  work  is  applicable  to  any  teleoperator  system  built  on  ATM.  Teleoperators 
are  often  master-slave  systems.  The  operator,  watching  a  visual  representation  of  the  worksite,  performs  a 
task  by  moving  the  master  as  if  it  were  performing  the  task.  The  time-vaiying  position  control  inputs  of 
the  master  are  sent  to  the  remote  site  as  time-varying  position  control  inputs  for  the  slave.  The  slave 
therefore  executes  a  scaled  copy  of  the  master’s  motion.  When  the  slave  contacts  the  environment, 
position  error  signals  and  contact  force  information  are  generated.  These  are  fed  back  to  the  master  and 
are  used  to  backdrive  its  links,  giving  a  sense  of  contact  with  the  remote  task.  Properties  of  the  ATM  link 
determine  how  successful  this  sense  of  contact  really  is. 

The  task  performance  is  improved  by  providing  the  contact  force  information  to  the  master.  When  the 
measured  contact  force  between  the  slave  and  the  environment  is  fed  back  to  the  motors  of  the  master,  the 
contact  force  is  said  to  be  “reflected”,  and  the  teleoperator  is  said  to  be  controlled  bilaterally[13].  The 
block  diagram  of  Figure  7  represents  a  bilateral  controlled  system.  The  operator  commands  a  velocity 
forward,  through  the  master,  communication  block,  and  slave,  to  the  environment.  Likewise,  the  force 
sensed  at  the  environment  is  transmitted  back  through  these  blocks,  to  the  human  operator. 
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Figure  7,  Block  diagram  of  bilateral  control  teleoperator 


The  research  on  bilateral  control  over  ATM  networks  is  significant  in  two  ways.  First,  the  bilateral  control 
with  time-varying  delay  remains  an  open  problem;  second,  ATM  switching  is  an  emerging 
communications  technology  that  is  well  suited  to  the  problem  of  remote  operations.  The  primary 
advantage  of  ATM  is  that  it  was  specifically  designed  for  simultaneous  data,  voice  and  video 
transmission:  precisely  the  characteristics  required  for  the  teleoperation  in  general.  No  work  on  bilateral 
control  over  ATM  has  been  published  in  the  robotics  field. 

Our  study  concentrates  on  two  aspects.  First  is  source  trafiic  characterization  in  ATM  networks,  including 
characterization  of  source  trafiic,  estimation  of  end-to-end  transfer  delay  boimd,  and  statistic  analysis  of 
the  delay  pattern.  Second  is  design  of  the  adaptive  bilateral  control  law,  which  includes  an  adaptive 
bilateral  control  law  dealing  with  the  unknown  but  fixed  latency,  and  an  adaptive  control  law  dealing  with 
time-vaiying  latency  with  known  pattern.  The  stability  of  the  bilateral  controlled  teleoperator  will  be 
proved  as  well. 


Ongoing  Work 

Although  we  have  made  good  progress  in  the  last  year,  there  are  still  many  things  to  be  done.  The  only 
major  components  still  missing  are  the  ATM  link  from  our  lab  to  the  OR  and  the  MKM  telerobotics 
interface.  However,  there  are  several  ongoing  efforts  to  improve  the  system. 

The  characterization  of  teleoperation  parameters  on  ATM  networks  will  be  an  important  effort  for  not 
only  this  project,  but  future  teleoperation  work  in  general.  We  have  one  Ph.D.  student  working  on  this 
topic  for  her  dissertation. 

We  are  anticipating  having  bidirectional  audio  between  the  remote  workstation  and  the  mock  OR/actual 
OR  by  31  August  1997. 

Safety  issues  are  crucial  in  a  human/robotic  system  such  as  the  IRNS.  While  we  have  done  one 
preliminary  study  on  the  relevant  safety  issues  and  a  study  on  the  fault  tolerant  capabilities  of  the  system, 
we  have  not  yet  incorporated  sufficient  safety  precautions  in  the  IRNS,  The  top  priorities  for  this  task  are 
to  implement  a  control  handoff  protocol  between  the  two  worksites  and  automatic  safety  volume 
calculation  for  the  force  reflection  system. 

As  discussed  previously,  the  force  reflection  system  should  be  improved.  Currently,  it  is  too  easy  to  cause 
the  robot  to  move  near  singularities,  causing  rapid  joint  motions.  While  this  is  usually  not  accompanied  by 
rapid  cartesian  motions  at  the  handgrip,  it  is  disconcerting  to  the  user.  It  is  also  difficult  for  the 
inexperienced  user  to  guide  the  robot  back  into  better  conditioned  areas  of  the  workspace.  Finally,  the 
motion  itself  is  not  as  smooth  as  we  would  like.  While  the  mechanical  design  of  the  Puma  is  partially  at 
fault  here  (high  friction,  backlash),  we  feel  that  smoother  motion  should  be  achievable. 

We  plan  to  continue  to  improve  the  IRNS  GUI.  Underlying  the  whiteboarding  system  is  the  ability  to 
overlay  graphics  onto  our  video  streams.  This  opens  the  possibility  for  more  complex  enhancements  such 


17 


^  /i  % 


as  overlaying  of  segmented  images  and  augmented  reality  elements  [14].  We  will  present  these  ideas  for 
enhancement  to  our  consulting  physicians  to  help  determine  which  should  be  implemented. 

Because  the  IRNS  is  still  under  development,  we  have  not  conducted  any  formal  experiments.  Now  that 
we  have  a  mostly  complete  implementation  we  plan  to  conduct  trials  with  our  consulting  physician  in 
order  to  take  measurements  on  network  loading,  operational  efficacy,  learning  time,  and  registration 
accuracy. 
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